test: add Anthropic Messages recording mode + gateway-tool cassettes#123
Conversation
Adds a `messages` mode to the cassette recorder for the Anthropic Messages API (/v1/messages), and records the upstream gateway-tool traffic that the Claude Code server-side tool loop (vllm-project#115) will replay in tests. The mode is stateless (resends full `messages` history each turn); with `--tool-outputs`, a turn following a `tool_use` feeds back matching `tool_result` blocks keyed by tool name, mirroring how a gateway tool loop feeds results back. Streaming reconstructs the final message from the Anthropic SSE (content_block_delta / input_json_delta / message_delta). Recorded against vLLM 0.25.1 (Qwen3-30B-A3B-FP8, hermes parser) serving /v1/messages natively — a two-turn web_search turn: model emits the gateway tool_use, tool_result is fed back, model answers. Both streaming and non-streaming. Reproducer: record_messages_cassettes.sh. These are fixtures for the (maintainer-gated) vllm-project#115 loop; there is no Rust consumer yet, since the Messages cassette loader lands with that loop. Part of vllm-project#115. Signed-off-by: Ashwin Giridharan <girida@amazon.com>
There was a problem hiding this comment.
@ashwing thanks. just to check my understanding from about the flow. So this PR is only adding cassettes file recorded from vllm upstream v1/messages entrypoint then in a follow up PR would add in the request/response payload mapping on gateway for messages api?
then in another PR handling the SSE chunk mapping?
|
Yes, exactly. This PR is just the recorder SSE/streaming chunk mapping I'd keep as its own piece rather than fold into #115 — the streaming cassette here already captures the Anthropic |
|
@maralbahari @franciscojavierarceo can we merge this to unblock the loop changes? |
When an Anthropic Messages request declares a gateway-owned tool (web_search), run the server-side gateway tool loop natively against vLLM /v1/messages — execute the tool, hide it from the client, and surface only the final assistant message. Requests without a gateway-owned tool keep the transparent proxy (vllm-project#99). The loop talks Anthropic Messages end to end (no RequestPayload/ResponsePayload detour), so the client's request is forwarded to vLLM untouched and every Anthropic field is preserved. It reuses only the protocol-neutral tool layer (ToolRegistry::dispatch, name-based classification, 60s per-call timeout, concurrent execution) via a small tool seam. - types/messages: Anthropic wire types + tool seam (tool_use<->FunctionToolCall, ToolOutput->tool_result, gateway/client classification). - executor::messages_loop: non-streaming loop (hide-the-call, feed tool_result back, round cap; client-owned tool_use returns to the client). - executor::messages_stream: streaming loop + MessagesStreamAccumulator that presents one logical message across rounds (single lifecycle, contiguous block indices, gateway tool_use suppressed) — the Anthropic-native analogue of the Responses GatewayAccumulator (vllm-project#119). - handler: routes gateway-tool requests to the loop, forwards the client's x-api-key, returns the Anthropic error envelope on failure. Tests: tool-seam + accumulator unit tests, non-streaming and streaming acceptance tests replaying the recorded vllm-project#123 /v1/messages cassettes, and handler routing tests. Validated live on G6e (Qwen3 + real You.com): non-streaming, streaming, mixed client+gateway tools, and proxy fallthrough. Implements vllm-project#115. Signed-off-by: Ashwin Giridharan <girida@amazon.com>
2 participants
Summary
Part of #115 (Claude Code — Stage 2). Adds a
messagesmode to the cassette recorder for the Anthropic Messages API (/v1/messages), and records the upstream gateway-tool traffic that the Claude Code server-side tool loop will replay in tests.The Stage 2 server-side loop itself is gated on making the executor loop API-agnostic (it's currently typed to Responses
RequestPayload/ResponsePayload) — that's a core refactor for a separate change. Recording the acceptance cassette has no such dependency and is the unblocked prerequisite #115 calls out, so this lands the recorder + fixtures ahead of the loop.What's here:
messagesrecorder mode (record_cassette.py): sends Anthropic/v1/messageswith tools, stateless (resends fullmessageshistory each turn). With--tool-outputs, a turn following atool_usefeeds back matchingtool_resultblocks keyed by tool name — mirroring how a gateway tool loop feeds results back. Streaming reconstructs the final message from the Anthropic SSE (content_block_delta/input_json_delta/message_delta).messages/): a two-turnweb_searchgateway-tool session — model emits the gatewaytool_use,tool_resultis fed back, model produces the final answer. Both streaming and non-streaming.record_messages_cassettes.sh+ a README modes-table row.Test Plan
/v1/messagesnatively),Qwen/Qwen3-30B-A3B-FP8, hermes tool parser + qwen3 reasoning parser.python -m py_compile record_cassette.pyclean;--helpshows the new mode;cargo build --workspaceunaffected (no Rust changed).POST /v1/messages, turn 1 emitstool_use(web_search)withstop_reason: tool_use, turn 2 request carries the full Anthropic history (user→assistantwithtool_use→userwithtool_result), turn 2 produces the finalend_turnanswer.<think>-taggedtextdeltas while non-streaming arrives asthinkingblocks — this is an accurate capture of what vLLM emits on each path, recorded intentionally so the future parser handles both.